Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[numpy] add new port #37409

Draft
wants to merge 49 commits into
base: master
Choose a base branch
from
Draft

[numpy] add new port #37409

wants to merge 49 commits into from

Conversation

Neumann-A
Copy link
Contributor

@Neumann-A Neumann-A commented Mar 13, 2024

Good to also have for this:

  • [blas/lapack] force vendor #24327
  • A Fortran compiler
  • More knowledge how cross builds of python work.
  • Fix arm64-windows cross builds
  • x86-windows cython not linking python lib

@JonLiu1993 JonLiu1993 added the category:new-port The issue is requesting a new library to be added; consider making a PR! label Mar 13, 2024
@Neumann-A
Copy link
Contributor Author

@m-kuhn:

Traceback (most recent call last):
  File "/mnt/vcpkg-ci/b/numpy/import_test.py", line 4, in <module>
    from numpy import *
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/__init__.py", line 149, in <module>
    from . import lib
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/lib/__init__.py", line 23, in <module>
    from . import index_tricks
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/lib/index_tricks.py", line 12, in <module>
    import numpy.matrixlib as matrixlib
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/matrixlib/__init__.py", line 4, in <module>
    from . import defmatrix
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/matrixlib/defmatrix.py", line 12, in <module>
    from numpy.linalg import matrix_power
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/linalg/__init__.py", line 73, in <module>
    from . import linalg
  File "/mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 35, in <module>
    from numpy.linalg import _umath_linalg
ImportError: /mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/linalg/_umath_linalg.cpython-311-x86_64-linux-gnu.so: undefined symbol: _gfortran_concat_string

Seems like the test needs to know about implicit link directories to setup LD_LIBRARY_PATH?

@m-kuhn
Copy link
Contributor

m-kuhn commented Mar 15, 2024

Just exportring LD_LIBRARY_PATH=${CURRENT_PACKAGES_DIR}/lib won't help here?

@Neumann-A
Copy link
Contributor Author

Just exportring LD_LIBRARY_PATH=${CURRENT_PACKAGES_DIR}/lib won't help here?

Nope. gfortran should be installed by the system on linux.

@m-kuhn
Copy link
Contributor

m-kuhn commented Mar 15, 2024

How is this supposed to be discovered at runtime (after installation)?

@Neumann-A
Copy link
Contributor Author

How is this supposed to be discovered at runtime (after installation)?

Was a missing -lgfortran in lapack. Don't know why numpy compiles with -Wl,--allow-shlib-undefined

@m-kuhn
Copy link
Contributor

m-kuhn commented Apr 3, 2024

Do I understand this correctly that this is currently blocked by static builds not having a python executable linked to cython?
Should there be a helper function vcpkg_acquire_python_with_modules(cython ...) that can be invoked by consumers (like numpy here) to create a python executable with specific additional modules linked?

@Neumann-A
Copy link
Contributor Author

Do I understand this correctly that this is currently blocked by static builds not having a python executable linked to cython?

Don't know if this is the problem. For me the problem is that I don't know how to create static python extensions which can be embedded into an application since that is what would be required for downstream users. (->https://www.kitware.com/static-python-and-numpy-with-paraview/)

Should there be a helper function vcpkg_acquire_python_with_modules(cython ...) that can be invoked by consumers (like numpy here) to create a python executable with specific additional modules linked?

Would be nice and is probably required if downstream tries to invoke python somehow with the modules for the build

@m-kuhn
Copy link
Contributor

m-kuhn commented Apr 4, 2024

Or this is listed as not supported static builds until this is sorted

@Neumann-A
Copy link
Contributor Author

@m-kuhn: Ideas how to solve the linux regression?
ImportError: /mnt/vcpkg-ci/p/numpy_x64-linux/lib/python3.11/site-packages/numpy/linalg/_umath_linalg.cpython-311-x86_64-linux-gnu.so: undefined symbol: _gfortran_concat_string
Somehow need to tell python that since lapack-reference is static it needs to load gfortran itself.

@m-kuhn
Copy link
Contributor

m-kuhn commented Apr 22, 2024

Do I understand this correctly: Numpy is built dynamic while lapack-reference (which contains _gfortran_concat_string) is static and python links lapack-reference?

If yes, I guess numpy needs to link lapack-reference too which might well result in ODR violations as lapack-reference is present in two places. Or lapack-reference and its dependency tree need to be built dynamic as well which should work.

@Neumann-A
Copy link
Contributor Author

Neumann-A commented Apr 22, 2024

@m-kuhn: No the problem is that the so does not know to load gfortran:

$ readelf -d _umath_linalg.cpython-311-x86_64-linux-gnu.so


Dynamic section at offset 0x3c6c80 contains 26 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x1b000

@Neumann-A
Copy link
Contributor Author

Ah wait that is just the same issue as before. I thought I already solved that one but the merge probably killed it.

@Neumann-A
Copy link
Contributor Author

@autoantwort:
Could you investigate the arm64-osx failure here?
For me it seems like the build is using the wrong (system) python for some reason since the file list clearly shows that the missing module should be present in the vcpkg installed python.

@autoantwort
Copy link
Contributor

[75/301] /Users/leanderSchulten/git_projekte/vcpkg/buildtrees/numpy/src/1d0eb3c352-64e3143c32.clean/numpy/_build_utils/tempita.py ../src/1d0eb3c352-64e3143c32.clean/numpy/random/_bounded_integers.pxd.in -o numpy/random/_bounded_integers.pxd
FAILED: numpy/random/_bounded_integers.pxd 
/Users/leanderSchulten/git_projekte/vcpkg/buildtrees/numpy/src/1d0eb3c352-64e3143c32.clean/numpy/_build_utils/tempita.py ../src/1d0eb3c352-64e3143c32.clean/numpy/random/_bounded_integers.pxd.in -o numpy/random/_bounded_integers.pxd
/opt/homebrew/opt/python@3.12/bin/python3.12
Traceback (most recent call last):
  File "/Users/leanderSchulten/git_projekte/vcpkg/buildtrees/numpy/src/1d0eb3c352-64e3143c32.clean/numpy/_build_utils/tempita.py", line 7, in <module>
    from Cython import Tempita as tempita
ModuleNotFoundError: No module named 'Cython'

And first line of buildtrees/numpy/src/1d0eb3c352-64e3143c32.clean/numpy/_build_utils/tempita.py

#!/usr/bin/env python3

ports/blas/portfile.cmake Outdated Show resolved Hide resolved
ports/blas/portfile.cmake Outdated Show resolved Hide resolved
ports/cblas/portfile.cmake Outdated Show resolved Hide resolved
ports/cblas/portfile.cmake Outdated Show resolved Hide resolved
ports/cblas/vcpkg.json Outdated Show resolved Hide resolved
ports/cblas/vcpkg.json Outdated Show resolved Hide resolved
@aminya
Copy link
Contributor

aminya commented May 21, 2024

This PR can possibly fix this issue too:
#34565

@Neumann-A Anything blocking you to make this ready? I might be able to help

@Neumann-A
Copy link
Contributor Author

This PR can possibly fix this issue too:
#34565

Figuring out cross-compiling with pyhton. Crossenv only works for linux and not for windows.

# Conflicts:
#	ports/lapack-reference/vcpkg.json
# Conflicts:
#	ports/vcpkg-tool-meson/meson.template.in
#	ports/vcpkg-tool-meson/vcpkg.json
#	ports/vcpkg-tool-meson/vcpkg_configure_meson.cmake
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:new-port The issue is requesting a new library to be added; consider making a PR!
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants